This specification relates to data processing and refining image relevance models.
An image search apparatus can use an image relevance model to evaluate relevance between textual user queries and images. For example, the relevance of an image to a user query can be determined, in part, based on textual content depicted in the image or textual content associated with the image, e.g., textual content appearing on a web page in association with the image. The relevance of an image to a query can also be determined by analyzing the way in which users who submit user queries interact with images responsive to the user queries. For example, an image search system may respond to a user query with one or more images, which are then displayed on a user device. If the user clicks on or otherwise interacts with one or more images, then this user interaction can be considered an indication that that those images were good (relevant) responses to the user query. If a particular image satisfies an image relevance threshold, e.g., a certain number of users click on the image when the image is presented as responsive to a user query, then the image may be labeled with a reference to the user query, e.g., a term included in the query, based on the determination that the user query accurately describes the image.
An image relevance model can be used to identify visual features of multiple images that have been identified as relevant to the same query. For example, multiple images that have been identified as relevant to the same query may have similar visual features, such as similar colors, brightness, shapes, edge locations, and/or other similar attributes. For example, images associated with the query “sunrise” will likely share similar colors (of a sky at sunrise) and shapes (of a sun appearing on the horizon). The attributes identified by the image relevance model can be used to further identify other images sharing the same common features—put another way, images characterized by the same common features—that may also be relevant to a query.